Providing scalable and efficient content delivery is becoming more important as the demand for applications such as streaming media is growing fast. Content Distribution Networks (“CDN”) and network service providers, advocate using network overlays for providing scalable and robust Internet based applications. Typical overlays are administrator configured, and due to the centralized nature of the overlay construction process, it is not feasible to construct large overlays.
Recent application-level overlay networks, such as CAN, eCAN, Chord and PAST, are scalable and self-organizing in nature. Nodes in these networks collectively contribute towards a fault-tolerant and administration-free storage space. The basic functionality these systems provide is a distributed hash table (“DHT”). In these systems, an object is associated with a key. Every node in the system is responsible for storing objects whose keys map to the ID of the node (via hashing). Retrieving an object amounts to routing to a node that is responsible for storing that object. The routing path on these overlay networks is at the application-level rather than at the IP level.
While elegant from a theoretical perspective, these systems suffer from at least two limitations. First, they rely on application-level routing that largely ignores the characteristics of the underlying physical networks. Because the underlying physical characteristics are not taken into consideration, excessive routing delays typically result. Second, they construct a homogeneous structured overlay network, while in reality, the nodes usually have different constraints and capacities such as storage, load, packet forwarding capacities and network connections.
In addition, overlay networks are typically constrained. In other words, the number of connections for a node is fixed or limited. Because of the constraints, the ability to accurately model the underlying physical characteristics is limited as well. Further, the earlier auxiliary networks do not handle the dynamic nature of the underlying network well, for example, when nodes exit or enter the network.
Still further, the amount of state information that needs to be maintained in the overlay network may be excessive.